Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation

Current speech translation systems integrate (loosely or closely) two main modules: source language speech recognition (ASR) and source-to-target text translation (MT). In these approaches, source language text transcript (as a sequence or as a graph) appears as mandatory to produce a text hypothesis in the target language. In the meantime, deep neural networks have yielded breakthroughs in different areas including machine translation and speech recognition. Current systems have an “encoder-decoder” architecture where a sequence of input symbols (or an input signal) is projected into a continuous low dimensional space and the output sequence is generated from this representation. This type of architecture has been proposed for machine translation (Sutskever et al., 2014; Bahdanau et al., 2015) and for automatic speech recognition (Chorowski et al., 2015; Chan et al., 2016; Zweig et al., 2016). If we exclude some pioneering attempts to directly translate from a source phone sequence to a target word sequence (Besacier et al., 2006) or to discover a bilingual lexicon from parallel phone-word sequences (Godard et al., 2016), the only attempt to translate directly a source speech signal into target language text is that of Duong et al. (2016). However, the authors focus on the alignment between source speech utterances and their text translation without proposing a complete end-to-end translation system. The same authors (Anastasopoulos et al., 2016) also propose to jointly use IBM translation models (IBM Model 2), and dynamic time warping (DTW) to align source speech and target text, but again, only the alignment performance is measured in their work.